Skip to content

fix: redact every column that names a user, verify unrecorded MySQL columns - #52

Merged
gjtorikian merged 5 commits into
mainfrom
fix/redact-user-references-and-unrecorded-columns
Nov 6, 2026
Merged

gjtorikian merged 5 commits into
mainfrom
fix/redact-user-references-and-unrecorded-columns

Conversation

@gjtorikian

Copy link
Copy Markdown
Collaborator

Summary

  • redact --user in Postgres, MySQL and MongoDB erased rows by user_id only. An invitation names a user as its inviter and as the user who accepted it, so those rows, and the archived and quarantined events carrying those payloads, kept the subject's id and email after the command reported success. Every column and payload field that names a user is matched now (user_id, inviter_user_id, accepted_user_id), the same fix the ClickHouse package got in feat(clickhouse): add the ClickHouse destination package #46.
  • MySQL recorded a column with no _columns record as current without a look at the table, so after a schema applied by hand from init --print-sql, a changed path was recorded as installed with the column still reading the old path. The catalog's clause now goes onto a temporary table and SHOW CREATE TABLE renders that and the live table with the same server, so the two compare as text; a match is recorded and anything else is drift that --repair rebuilds. One pinned connection, since the temporary table lives in its session, and within the CREATE TEMPORARY TABLES grant the README already lists. Postgres already did this through a temp table and pg_get_expr.

Testing

Each package's redaction test now creates an invitation the subject accepted, one the subject sent and one unrelated, and expects two invitation rows erased, one left, and no archived event naming the subject. MySQL's drift suite covers the unrecorded column: the same catalog checks out and is recorded, a changed path is reported as drift and withholds the fingerprint, and --repair rebuilds it. The three suites pass locally (Postgres on PGlite, MySQL 8.4, MongoDB memory server).

…olumns

`redact --user` in Postgres, MySQL and MongoDB erased rows by `user_id`
only. An invitation names a user as its inviter and as the user who
accepted it, so those rows, and the archived and quarantined events
carrying those payloads, kept the subject's id and email after the
command reported success. Every column and payload field that names a
user is matched now, as in the ClickHouse package.

MySQL recorded a column with no `_columns` record as current without a
look at the table, so after a schema applied by hand a changed path was
recorded as installed with the column still reading the old path. The
catalog's clause now goes onto a temporary table, and `SHOW CREATE
TABLE` renders that and the live table with the same server, so the two
compare as text: a match is recorded, anything else is drift.
@greptile-apps

greptile-apps Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The PR appears safe to merge; the previous event-deletion issue is fixed.

Summary

Expands user deletion across all four destinations and checks MySQL columns that have no saved definition.

  • User redaction now follows every declared user reference.
  • MySQL init checks unrecorded generated columns against the live table.

Reviews (4) · Last reviewed commit: "test(engine): expect the catalog's user ..." · Reviewed by Greptile

Comment thread packages/sync-mysql/src/schema/drift.ts Resolved
…tion

Which columns hold a user id is a fact about the catalog, not about any
database, yet each destination carried its own list. A column can now
declare what it references (`user_id` references `users`), and the
engine answers `userReferenceColumns(stream)` for the mirror rows and
`userReferenceFields()` for the archived payloads; the four `redact`
implementations ask it instead of repeating the list.

The MySQL README's "no DDL after the first init" promise now says what
checking an unrecorded column does: a temporary table created and
dropped within the session, which is session state, not a schema change.
Comment thread packages/sync-engine/src/catalog.ts Resolved
Comment thread packages/sync-engine/src/catalog.ts Resolved
…om the streams

An extra column named `user_id` counted as naming a user before and
still does, without declaring anything. The payload paths `redact`
scrubs archived and quarantined events by come from the effective
streams, extra columns included, as path steps rather than top-level
field names, so a nested reference such as `metadata.owner_user_id`
is scrubbed too. Each destination builds its JSON path from the steps:
a parameterised array for Postgres, the JSON path helpers elsewhere.
Comment thread packages/sync-engine/src/catalog.ts Resolved
…treams are selected

The archive keeps every event type the tail subscribed to, so a run
mirroring only `users` still archives `invitation.accepted` with the
subject's id in it. The payload paths `redact` scrubs by now come from
the whole catalog, with the selection adding only its extra columns.
@gjtorikian
gjtorikian merged commit 613ae96 into main Nov 6, 2026
23 checks passed
@gjtorikian
gjtorikian deleted the fix/redact-user-references-and-unrecorded-columns branch November 6, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant